1. Technical Field
One or more embodiments of the present invention generally relate to networks supporting remote direct memory access. More particularly, certain embodiments relate to networks with a constrained usage model.
2. Discussion
On a network with multiple nodes (for example, computers), an application on one of the nodes may generate a request to read or write to upper-level (for example, application-level) memory on another node of the network. These remote read and write operations generally result in data being transferred from one node to another over the network. In designing networks, one of the considerations has been the speed at which these remote read and write operations may be performed. However, in some cases, these remote read and write operations involve making kernel or other intermediate copies of the data being transferred. The making of intermediate or kernel copies generally slows remote read and write operations, which increases latency.
Remote direct memory access (RDMA) allows remote reads and writes to be performed on a network, while minimizing the making of intermediate copies of the data being transferred between nodes. In some implementations of RDMA, reads and writes involve establishing a connection over the network between a queue pair on a first node and a queue pair on a second node. A queue pair generally comprises at least a send queue (SQ) and a receive queue (RQ). Then, data being sent from a node is sent from the SQ and data being received by a node is received at the RQ. The queue pair may be referred to as a work queue. Application requests to access remote memory may be placed in a queue pair as work queue events. Other types of queues may also be present. For example, there may be a completion queue to communicate the completion of work queue events.
Some RDMA packets may carry data and others (for example, a read request) may not. An RDMA data packet flowing from one node to another may be referred to as a protocol data unit (PDU).
One of the design issues for networks implementing RDMA is the architecture of both the network itself and the nodes on the network. Another design consideration is the protocol that is used over the network to move data from one node to another. Both the architecture and the protocol used may influence the speed and the reliability of remote reads and writes over the network.